The documentation is not available anymore as the PR was closed or merged.

I used this Colab to test the new push_to_hub on a large dataset (55 GB). It works great.

One thing that could be improved is the performance of dataset.data.nbytes - it takes ≈ 3 minutes to compute for the dataset in question (50k array chunks per column). It probably makes sense to store larger chunks locally. But this can be addressed in a subsequent PR.

@lhoestq

single commit can fail (time out) if there are too many operations so we might have to do multi commits anyway in that case

Multiple commits complicate the logic significantly. Maybe, let's keep things simple and emit a warning if there are more than 100 additions (we can suggest increasing max_shard_size in that case). Additionally, we can set the default max_shard_size to a higher value, e.g., 5GB. I think handling up to 500GB of data in the default case seems reasonable. In rare cases where this is a problem, one could increase the default max_shard_size even further (if RAM is not a limiting factor) or use to_parquet + huggingface_hub (we could have a docstring or a doc note that explains this).

Note that we split the dataset based on the Arrow data size, which means Parquet shards will be considerably smaller unless there are binary fields such as image JPEGs in the dataset, which are hard to compress efficiently.

how to let users resume a push_to_hub that failed mid-way because of a connection error for example

They can resume by rerunning the failed push_to_hub.

preupload_lfs_files will be instant in that scenario, as explained in huggingface/huggingface_hub#1699 (comment)

Multiple commits complicate the logic significantly. Maybe, let's keep things simple and emit a warning if there are more than 100 additions (we can suggest increasing max_shard_size in that case)

I don't think we can do that, many people are uploading files with 100+ files and it would break their workflow

Well done! 👏 🔥

Congrats on merging this! 👏